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Abstract 

We study the influence of stochastic effects due to finite population size in the evolutionary 
dynamics of populations interacting in the multi-person Prisoner's Dilemma game. This paper 
' is an extension of the investigation presented in a recent paper [Eriksson and Lindgren (2005), 

fNj . J. Theor. Biol. 232(3), 399]. One of the main results of the previous study is that there 

are modes of dynamic behaviour, such as limit cycles and fixed points, that are maintained 
. due to a non-zero mutation level, resulting in a significantly higher level of cooperation than 

was reported in earlier studies. In the present study, we investigate two mechanisms in the 
evolutionary dynamics for finite populations: (i) a stochastic model of the mutation process, 
and (ii) a stochastic model of the selection process. The most evident effect comes from the 
second extension, where we find that a previously stable limit cycle is replaced by a trajectory 
P_| ' that to a large extent is close to a fixed point that is stable in the deterministic model. The 

Oh . effect is strong even when population size is as large as 10 4 . The effect of the first mechanism 

q ' is less pronounced, and an argument for this difference is given. 

• i— i 

1 Introduction 

When several individuals interact in a group to produce a good or to receive a benefit, the best 
strategy for a player depends on the details of the game as well as on the strategies of the others. 
For the group as a whole, and often for each individual in the long-term perspective, it would 
be best if cooperation could be established. In social and natural systems, there are, though, 
numerous examples of situations where so-called free riders or defectors take advantage of others 
cooperating for a common good [1-3]. A game-theoretic approach for the study of cooperation 
can be based on the Prisoner's Dilemma game [4, 5] - a situation that captures the temptation to 
"q I act in a selfish way to gain a higher own reward instead of sharing a reward by cooperating. In the 
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game, the players independently choose an action, either to defect or to cooperate. 

In the two-person game, the scores are R (reward) for mutual cooperation, T (temptation 
score) for defection against a cooperator, S (sucker's payoff) for cooperation against a defector, 
and P (punishment) for mutual defection, with the inequalities S < P < R < T and (usually) 



^ ■ R > (T + S)/2. We use fixed values on R and S in this study, R = 1 and S = 0, while < P < 



1 < T < 2; in the population dynamics we use there are only three independent parameters, the 
third one being a growth constant. From theoretic and simulation studies of two-person Prisoner's 
Dilemma game, it is known under which circumstances repeated interactions may allow for a 
cooperative population to be established that can resist invasion by non-cooperative mutants (see, 
e.g., [5-11]). 

In the ?i-person Prisoner's Dilemma game, n players simultaneously choose whether to coop- 
erate or to defect. In the literature, there are several evolutionary models based on the n-person 
Prisoner's Dilemma using various strategy sets and pairing mechanisms, e.g., where the players are 
distributed in space (see, e.g., [12-19]). In a recent paper [20], we revisited the classic n-person 
Prisoner's Dilemma. Following Boyd and Richersson [21] and Molander [22], the behaviours of 
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the participants were modelled by simple reactive strategies. These authors analyse the stability of 
stationary populations in the limit where mutations are infrequent: a mutation either is driven to 
extinction by the selective pressure from the resident population, or leads to a new resident pop- 
ulation. The general conclusion from their studies is that cooperation is difficult to obtain when 
extending the group size beyond the two persons in the original Prisoner's Dilemma game. In this 
limit, we have found [20] that for some values of the payoff parameters, the rate of convergence to 
the evolutionarily stable populations is so low that the assumption that mutations in the population 
are infrequent on that time scale is unreasonable. Furthermore, the problem is compounded as the 
group size is increased. In order to address this issue, we derived a deterministic approximation of 
the evolutionary dynamics with explicit, stochastic mutation processes, valid when the population 
size is large. 

The question is: does the deterministic replicator dynamics introduced in [20] accurately de- 
scribe the time evolution of the frequencies of strategies in a finite population? In the present 
study, we investigate two mechanisms in the evolutionary dynamics for finite populations: (i) a 
stochastic model of the mutation process, and (ii) a stochastic model of the selection process. 

2 The n-person Prisoner's Dilemma game 

In the n-person Prisoner's Dilemma game each player interacts with n — 1 other players. Depend- 
ing on the number k of others cooperating, a player receives the score V(C\k) when cooperating 
and the higher score l/(D|/c) when defecting. In order for the model to be well-behaved, the score 
functions must obey two constraints: first, the scores must increase with an increasing number 
of cooperators. Second, the sum of the scores given to all players should increase if one player 
switches from defection to cooperation (see, e.g., [21]). In this paper we shall assume that the 
scores V can be calculated as a linear combination of the scores against the other players in n — 1 
ordinary two-player Prisoner's Dilemma games: 

V(C\k) = J^ a ndV(D\k)=T^- + p n ~ k ~ 1 , (1) 
n — 1 n — 1 n — 1 

where we have divided by n — 1 in order to make it easier to compare results from different group 
sizes. The parameters P and T obey 0<P<1<T<2. Note that this is still an n-person 
game since the same action is performed simultaneously in all games. It is straight-forward to 
extend this model to arbitrary score functions V(C | fc) and V(D|/c) (provided the above constraints 
are fulfilled). The qualitative conclusions of this paper, however, are not expected to depend on 
the choice of score functions. 

We focus on the set of trigger strategies [23] as the strategy space for the evolution, which was 
also considered by, e.g., Boyd and Richersson [21] and Molander [22]. Despite their simplicity, 
trigger strategies capture many important aspects of the many -person game, and allow for straight- 
forward evaluation of the expected score for a player in a group randomly generated from a given 
population. A trigger strategy is characterised by the degree of cooperation that it requires 
in order to continue to cooperate: a player with trigger strategy Sk cooperates if at least k other 
players cooperate. In a game with n participants, k is in the range 0, . . . , n. The strategy sq 
is an unconditional cooperator and s n is an unconditional defector. Each player decides whether 
to cooperate or to defect based on the actions of the other players. In the first round after the 
formation of a group, all players are assumed to cooperate, with the exception of unconditional 
defectors. Then the players that are unhappy with the number of cooperators switch to defection. 
This may cause other players to change their action, and this is iterated until a stable configuration 
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has been reached. Note that the number of cooperators may only decrease or be stable, and that 
this procedure converges to the stable configuration with the maximum number of cooperators. In 
a repeated game without noise, this implies that a group of players with different trigger levels 
reaches a certain degree of cooperation, some players may be satisfied and cooperate while the 
others defect. We use the scores for the players in this equilibrium state to determine the selection, 
described in the next section. 

3 Evolutionary dynamics 

Consider a population of N individuals. From one generation to the next, a fraction 5 of the 
population is replaced using fitness proportional selection, where the fitness of an individual is 
proportional to the number of offspring surviving to reproductive age. Throughout this paper, 
5 = 0.1. If small enough, the value of 5 does not influence the structure of the evolving population, 
but determines the evolutionary time scale. Assuming that the population size is large and constant, 
the evolutionary dynamics takes the form of 

x\ = xi + 5 f j - lj xi , (2) 

where x\ is the fraction of players in the population with trigger level I, x\ is the value of xi in the 
next generation, fi is the expected fitness for a player with trigger level I, and / = Yl?=o x ifi * s 
the average fitness in the population. The expected fitness /; for a player with trigger level I is the 
expected score of the player in a randomly formed group: 

n 

fl = Yl • • • x in _ 1 S"(/,n, . . . ,i n _i) (3) 

h, i«-i= 

where S(l, i±, . . . , i n -i) is the score of a player with strategy s; in a game with n — 1 other players, 
using strategies Sj i; . . . , Si n _ 1 respectively. 

Molander [22] has analysed this model - with general score functions V(C\k) and V(D|fc) - 
under the assumption that a mutation will either lead to a new resident population, or that the 
evolutionary dynamics © will bring the population back to the original population before the 
next mutation occurs. Molander has shown that in each interval P E (^Ei> ^t)> where k € 
{1, . . . , n — 2}, there is either a mixture of strategies s*. and s n , which is evolutionary stable, or 
there is a mixture of strategies so, . . . , s n -i (all cooperating), that resists invasion by strategy s n , 
but which is not evolutionary stable. Finally, there is no other asymptotically stable population 
in that interval. In the interval P £ (SEt j 1)> tne P urei y cooperative equilibrium mixture is the 
only possible asymptotically stable population. 

Consider a population with groups of size n consists of a mixture of strategies and s n , in 
fractions x and 1 — x, respectively. Since, in this population, strategy Sfe cooperates if and only if 
there are at least k other players with the same strategy in the group, and since strategy s n always 
defects, direct evaluation of © gives 

/*(*) = i2 p ( n i Ziy~ 1 ( 1 - x ) n ~ i + E ^i(iI 1 1 ) x< " 1(1 - a:)B " i (4) 

1=1 v J i=k+l v y 
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Figure 1: The payoff at equilibrium as a function of P (thick line), for T = 1.2 and group size 
n € {2, . . . , 10}. Also shown is the payoff P (dashed line) for a population of pure all-defect. 
(Adapted from [20].) 
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(5) 



for strategy s n . We find the equilibrium by setting fk{x) = f n (%) an d then solving numerically for 
x, with the requirement < x < 1. Existence and uniqueness of this equilibrium is guaranteed 
by the result of Molander [22]. In Fig. Q we show how the equilibrium fitness depends on P 
for T = 1.2, for group size n € {2, . . . , 10}. The fitness at the asymptotically stable population 
approaches / = P as n increases, indicating a decreasing degree of cooperation with increasing n. 
From the existence and uniqueness of the asymptotically stable populations of this form, and from 
a Taylor expansion of fk and f n to the k + 1th order, follows that xu oc 1/n at the asymptotically 
stable population, for T > 1. The decrease in cooperation in the evolutionarily stable populations 
is what has led earlier authors [21,22] to conclude that cooperation in large or event modestly 
sized groups is evolutionarily unfavoured. 

The number of terms in direct evaluation of the expected fitness from © grows very rapidly 
with n. Hence, in order to simulate the evolutionary dynamics $2% for general population com- 
positions, it is necessary to have an efficient method for evaluating the expected fitness of each 
strategy. Using the probability P\ that the number of cooperating players equals i, in a group with 
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one player using strategy si and the n — 1 other players chosen randomly from the population, we 
may express the expected fitness fi as 



fi = E^ y (°l i )+ E PiV(C\i-l), (6) 

i=0 i=l+l 

since strategy si cooperates if i > I. Thus, efficient calculation of the P\ allows for the study of 
the evolutionary dynamics in groups of more than a few players. Since P\ only depends on the 
distribution of trigger levels in the population, this method may be applied for any payoff functions 
V(C|i) and V(D|i). In the previous paper [20] we derived the following formula for P\: 

whenever i = I, 

D l '\ when i = 0, 

P = < _i (7) 

(xq + . . . + x n -i) n when i = n, 

{iXJ {^ + --- + Xi-ir 1{l<i) D l li-i +Hl<i) otherwise, 
where 1(/ <i) is one if I < i and zero otherwise, and Dm is given by the recursive formula 



x m when i = n — 1 

3=0 



where M = m + i + 2 — 1^ < ^ — n. Note that whenever / < i and I' < i, P- = Pj , so if I < i 
then P\ = Pf. Using tables to store evaluated values of Dm, it is possible to evaluate the values 
of P\ for all I and i in ~ n 4 operations. 



4 Selection and mutation processes in finite populations 

In [20] we introduced a simple model for incorporating mutations as an explicit part of the evolu- 
tionary dynamics. The population is subject to selection as in ©. In addition, a number M-i^j of 
individuals per generation switch from strategy si to strategy Sj due to mutations. The mutations 
are assumed to be generated by uncorrected stochastic events, e.g. by a Poisson process, in the 
process of reproduction. The evolutionary dynamics then takes the form 

x'i = Xi + sf^-T^Xi + ^^Mj^i-Mi-.j). (9) 

In a single generation, each player with strategy Sj has an expected number SfiXi/f of offspring 
surviving to reproductive age. Mutations occur independently in the creation of each offspring, 
with a probability of \i <C 1 per offspring per generation, and the strategy of the mutated offspring 
is chosen randomly among the n + 1 strategies, with equal probability. Since the population 
size is assumed to be large, we approximate the number Mi^j of mutated offspring with its 
expected value, M.i^j ~ fi5N fiXi/[(n + 1)/]. Inserting this approximation into (J9ji, we obtain 
the following expression for the evolutionary dynamics: 

Xi = Xi + 5 ( -= - 1 J Xi + 5(1 ( ^_ - - y Xi J . (10) 
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Figure 2: a The time-averaged average payoff in the population as a function of P, for n = 6, 
T = 1.6, and fi = 10~ 4 . Also shown is the payoff in the limit of infinitesimal /i (thick line) and 
the payoff P for a population of pure all-defect (dashed line). (Adapted from [20].) b The initial 
transient and two cycles of the attractor of the time evolution of the population for P = 0.33 
(Adapted from [20]). Within a cycle, strategy comes to dominate the population for a period 
of time. During the part of the cycle where strategy sq is prominent, the system approaches the 
population mixture which is evolutionarily stable when mutations are infrequent. For the value of 
P used here, it corresponds to a mixture of strategies S2 and sq. For a period of time, this mixture 
of strategies dominates the population. After a while, however, x^ starts to grow at the expense of 
xq, and after a while there is another sharp transition where £5 and £2 grow and xq decay to very 
low levels. 



For some values of the payoff parameters, the strength of selection is so small compared to the flow 
mutants that the population does not converge to the evolutionary stable composition of strategies 
in the population. This is illustrated in Fig. Ek, where we show the asymptotic time-averaged 
average fitness in the population, for case of infrequent mutations [22] and for (fTOl) . For some 
values of P the difference is clear; in this case, the dynamics either converge to a limit cycle 
(illustrated in Fig. |2t>X or to a fixed point where three or more strategies coexist at significant 
levels. 

In this article, we compare the results of the deterministic approximation (flOl to explicit sim- 
ulation of ©. It is natural to assume that in the stochastic model, the mutations in individuals 
with strategy Sj occur according to a Poisson process with rate Aj = fi5NfiXi/[(n + 1)/]. In 
each simulation step, the number of mutants of each strategy is drawn independently. For each 
mutation, the resulting strategy is drawn with uniform distribution and the fraction of the chosen 
strategy is incremented. 

An additional source of stochastic fluctuations come from the selection of individuals from 
one generation to the next. The growth of the fraction Xi of strategy Sj in (0) may be viewed as 
a mean-field approximation of a more realistic stochastic growth process, in which the difference 
in fitness between two individuals is reflected in the probability distribution of their number of 
offspring. For simplicity, we model mutations as occurring separate from the selection process 
- in our computer simulations, we first generate the next generation according to the selection 
model, and then impose the effect of mutations as described above. 

The selection process is as follows: in each generation, an individual with strategy s produces 
a number of offspring in relation to the fitness of strategy s. It is assumed that the total number of 
offspring is more than enough to replace the parent generation. The fraction of the offspring with 
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strategy Sj is then 



(1 ~ s)x,+ S 7 X " 



<ii) 



which we recognize as the fraction of Sj in the next generation in the case of an infinite population 
size. Since the environment has a finite carrying capacity, we sample N individuals uniformly 
from the offspring of the parent generation. Hence, the joint distribution of the number k{ of 
players with strategy Sj in the next generation is multinomial: 



/Oo, ... ,k n ) 



Nl 



n 



n 



iH - 



(1 - 6)xi + 5 



k 

I 



"I 



(12) 



provided the constraint Y17=o ki = N is fulfilled. In the limit of N — > oo, the variance of 
x\ = ki/N vanishes, and the dynamics becomes deterministic, as expected. 

The selection process can be approximated, for large populations, with a diffusion process: to 
each fraction X{ we add a normal distributed number f; with zero mean and a variance &Ej(l — 
xi)/N. We take the variables fj and £j to be uncorrelated, for all i ^ j. If a fraction Xi becomes 
negative, it is set to zero. Finally, the fractions are normalised so that Y17=o x i = This approx- 
imation does not exhibit the correct coiTelations in the fluctuations of the xi, but the magnitude 
of the fluctuations are approximately correct. Since the magnitude of the perturbations are small, 
however, coiTelations and higher-order cross-terms may be neglected. 



5 Results 

In Fig.|3]we illustrate the effect of random mutations and the finite population size on the evolution 
of the frequencies of the strategies, for a specific choice of the parameters. The effect is similar for 
other parameters; for parameter values where the deterministic approximation (flOt converges to a 
fixed point (c.f. Fig.|2k), introducing a finite (large) population size does not change the qualitative 
properties of the long-term evolutionary dynamics, although the variance of the fraction of each 
strategy is increasing with decreasing population size. 

We now focus on the fate of the limit cycles that are the most striking feature in the deviations 
of (fTTl from the evolutionary dynamics in the limit of infinitesimal mutation rate. The simulations 
show that, in the limit of very large populations, the analytical approximation dlOb of how the flow 
of mutants alter the replicator dynamics is valid. As can be see from the smoothness of the curves 
in Fig.|3k, the main effect of the stochastic mutations compared to their deterministic counterpart 
is to add a little noise to the time evolution so that the curve fluctuates around the deterministic 
solution. Hence, unless the population size is small, the difference between the explicit mutations 
and the analytical approximation is negligible. 

Fluctuations in the frequencies of strategies in the population due to random sampling in the 
selection play a very different role. Panels b-d in Fig. |3] illustrate the effect of these fluctuations 
for three values of the populations size. For very large populations (N ~ 10 8 ), the fluctuations 
do not alter the qualitative properties of the time evolution (panel b). However, for populations as 
large as N = 10 6 individuals (panel c), we find that the fluctuations cause significant deviations 
from the deterministic model, while some qualitative aspects are the same: the time evolution 
still exhibit cycles, characterised by short periods where the composition is similar to that of the 
infrequent-mutation solution of Molander [22], interspersed with long periods of a high degree of 
cooperation and with a slow drift of strategies. The period of these cycles now fluctuate. When 
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Figure 3: Illustration of the effect of stochastic mutations and a finite population size on the 
evolution of strategies. The parameters are: T = 1.6, P = 0.33, fi = 10~ 4 and the groups size 
n = 6. The lines are: sq (blue), si (green), S2 (red), S3 (cyan), S4 (magenta), S5 (yellow), and 
sq (black). In panel a we show the time evolution of a population with stochastic mutations but 
ignoring the effect of finite population size in the selection (population size N = 10 4 ). In panels 
b, c, and d, both explicit mutation and selection is present. The effective population size N takes 
the value 10 8 , 10 6 , and 10 4 , respectively. 

the population size is decreased to N = 10 4 , the cycles are no longer stable. The population now 
switches between irregular cyclic behaviour and more stable configurations in which three or more 
strategies co-exist. Preliminary investigations indicate that these configuration may have a stable 
counterpart in the deterministic approximation; it is not yet clear, however, if this he case for all 
parameter values. 

6 Discussion and conclusions 

The main purpose of this article (and also of the previous article [20]) is to better understand how 
more realistic models of the evolution of a population affects the evolution of the frequencies of 
strategies in the population. In the present article we have confirmed that - when the population 
size is large and the mutation rate is small - the deterministic approximation HOI provides an 
accurate description of the evolutionary dynamics. We have also shown that taking the fluctuations 
in the selection process into account may lead to different compositions of the strategies in the 
population than expected from the earlier analysis. 

The fluctuations in the selection of individuals show a much stronger effect on the evolutionary 
dynamics than does the fluctuations from the mutation process. A large part of this difference can 
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be understood from the magnitudes of these fluctuations, as characterized by their variances: in the 
mutation process, the variance contributed per time step is of order [5fiXi/(n + 1)]A^ _1 , whereas 
in the selection process the variance is 5xi(l — x%)N . When the mutation rate is small, and X{ is 
not very close to one, /z/(n+ 1) <C 1 — X{. For instance, when /i = 10~ 4 , N = 10 4 , and n = 6, as 
in Fig.|3k, the typical fluctuations due to mutations are of the same order as the fluctuations from 
the selection process in population with N ~ 10 8 individuals. This explains the good qualitative 
agreement between panels a and b in Fig. |3] 

It remains to explain what causes the evolutionary dynamics to switch from the limit cycle to 
coexistence of three strategies in Fig. |3Ji. A hypothesis is that, apart from the limit cycle, there 
are stable fixed points with a small basin of attraction when the population evolves under 111 Ob : 
the fluctuations in the selection process may then cause a transition from one mode to the other 
and back. In this case we expect that the transitions between these modes occur, approximately, 
according to a Poisson process. The number of such fixed points, and to which extent the proposed 
mechanism can explain the observed dynamics, remains to be investigated. 
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